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VARIATIONAL PDE-BASED MODELS FOR IMAGE 
FILTERING AND INPAINTING 
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Abstract. This work represents a survey of the main image denoising and inpainting techniques 
based on variational (enery-based) schemes that lead to nonlinear partial differential equation 
(PDE) —based filtering models. The variational denoising and restoration approaches are described 
first. Then, the energy-based structural image reconstruction models are surveyed here. Some of our 
own variational techniques proposed in these closely related image processing and analysis 
domains, will be also briefly described in this article. 
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1. Introduction 


Image noise removal and inpainting represent important and still-challenging 
image processing fields that are also closely related. An overview on these 
domains is provided in this research work. 

The image denoising and restoration represents the process of eliminating the 
electronic noise, which is a random variation of color information or brightness, 
from the 2D image signal [1]. There exist many types of image noise, the most 
popular of them being the 2D additive white Gaussian noise (AWGN), which 
represents a statistical noise having a PDF equal to that of the normal distribution 
and its major source arises during the acquisition and transmission processes. 
Finding an effective noise reduction method still constitues a challenge in the 
image processing area. Such a filtering scheme has to optimize the trade-off 
between noise removal, detail preservation and avoiding undesired effects, like 
blurring or staircasing. 

The classic denoising models, such as 2D Gaussian or Averaging filters, generate 
the blurring that corrupts the edges, corners and other features [1]. Therefore, the 
partial differential equation (PDE) - based filtering algorithms were introduced to 
solve properly this issue. The nonlinear PDE-based techniques have proved to be 
a great solution for this filtering task, providing effective detail-preserving 
restorations [2, 3]. They perform a directional diffusion that is degenerate along 
the gradient direction and has a smoothing effect along but not across the image 
boundaries. Some of these nonlinear diffusion-based models follow variational 
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principles and can be represented in energy-based form. They are disscused in the 
next section, where some of our contributions in this area are also described. 

The image interpolation (inpainting) aims to reconstruct the missing or highly 
damaged image regions as plausibly as possible, using the information extracted 
from the surrounding zones. This domain has the origin in the ancient art 
restoration [4] and has many applications, like digital artwork reconstruction, 
photo/movie renovation, object removal, image super-resolution/zooming and 
image compression. 

The inpainting methods can be grouped into 3 categories: texture-based, structure- 
based and combined schemes. The textural interpolation approaches are related to 
texture synthesis [5, 6], or represent exemplar-based algorithms [7]. The structural 
inpainting techniques employ information around the missing component in order 
to estimate the isophotes from coarse to fine, and diffuse that information. They 
complete the missing image region using variational and PDE-based algorithms 
[4]. The state of the art variational structural interpolation methods are presented 
in the third section. We have performed much research in the variational 
denoising and structure-based inpainting area, developing a lot of novel 
techniques [8]. Our main contributions to this research field are also described in 
these sections. This work ends with a conclusions section and the references. 


2.. Energy-based Noise ReductionTechniques 


The PDE-based denoising models following the variational principle are obtained 
from minimizations of various energy cost functionals. Such a energy functional 
sums a regularization to a fidelity component. The generic variational (energy- 
based) filtering scheme is given by minimization min {E(w) = R(w)+ FW}, 


where E(u) is the energy functional, the regularization term R(uw) =a@ i) y(\Vul)jaa, 
Q 


y is the regularizer function, and the fidelity term F(t) = £ f (ur /, y dQ. 
a 


wih a, B>O. 

Many such variational denoising models have been constructed in the last 30 
years. They lead to some nonlinear PDE-based models by applying the Euler- 
Lagrange equation and next the steepest gradient descent method. 

An influential energy-based restoration technique is that introduced in 1992 by 
Rudin, Osher and Fetami [9]. Their Total Variation (TV) - ROF Denoising model 
that minimizes the TV norm is very effective at simultaneously preserving the 
isophotes while filtering away the additive noise in the flat image regions. It has 
the following form: 
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u- =arg min [Iu + (to —uy? Jace (1) 
uel? (Q) Q 2A 

where A> O and u’* represents the restored image. A second-order nonlinear 

PDE model is then obtained, using the Euler-Lagrange equation and gradient 


descent [9], as follows: 
Mt as{ Mies LG o) (2) 


This total variation regularization technique removes properly the additive 
Gaussian noise, but it can be also adapted to other additive noise models. So, the 
TV Denoising model for the papacy noise is the following [10]: 


u= pa | (Ia + Ale U|)dQ (3) 


prs sientet 


where BV(Q) is the bounded variation image space. The total variation-based 
filtering model for Poisson noise is the next one [11]: 

u * =argmin | ( |\Vul+A(u= Uy logu))dQ (4) 

uEeBV (Q) OQ 
Many well-known second-order nonlinear anisotropic diffusion-based restoration 
models can be also described in variational form. Such an influential PDE-based 
restoration technique is the anisotropic diffusion scheme of Perona and Malik 
[12], expressed as: 
Ou . 2 

ae div(¢ (vu ). vu) (5) 


u(O, x, y)= Uy (x, y) 
where the diffusivity (edge-stopping) function g: [o, co] —> [0, co] iS monotonic 
decreasing, convergent to zero and g (0) = 1. The authors provided 2 such 


s2 


functions: ¢ (s?)=e 3 ¢(s?)= , k > 0. They also proposed a 


Oe 


numerical approximation for this PDE model, based on the finite difference 
method [12]. The P-M technique produces effective additive noise reduction 
results and preserves the boundaries very well. In the last decades a lot of 
anisotropic diffusion-based filtering models following the variational principle 
have been derived from this Perona-Malik approach [2,3]. 

Although the second-order nonlinear diffusion models, in both PDE and 
variational form, provide good detail-preserving denoising and deblurring results, 
they still have their own drawbacks. Such a major shortcoming is the unintended 
staircase effect that may create flat regions separated by artifact boundaries. So, 
many variational algorithms improving the total variation regularization model 
and overcoming this undesired effect have been proposed in the last 25 years. 
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Thus, the Total Variation Denoising with Split Bregman applies the iterative Split 
Bregman algorithm for solving the TV minimization problem [13]. An improved 
variant of the TV-ROF scheme is the TV — /; restoration model that is contrast 
invariant and able to separate the features according to their scales [14]. The 
Adaptive TV Denoising represents another improved version of the total variation 
regularizing [15]. Its regularizer component works like a total variation norm at 
the boundaries while approximating the /2- norm in flat and ramp regions in order 
to overcome the staircase effect. Higher Degree Total Variation (HDTV) model 
proposed by Hu and Jacob [16], constitutes another variational denoising solution. 
It minimizes properly the staircasing and ringing artifacts that are quite common 
to the TV-based schemes, applying new isotropic and anisotropic high-degree TV 
regularization penalties. The Generalized Total Variation (GTV) regularization 
model for image denoising, which is a generalized p™ power total variation, 
represents another improved version of TV-ROF Denoising [17]. 

Unlike the second-order PDE models, the nonlinear fourth-order PDE-based 
schemes produce piecewise planar images that look more natural and also avoid 
the staircasing. A very influential nonlinear fourth-order PDE denoising model is 
the isotropic diffusion-based algorithm of Y. -L. You and M. Kaveh [18]. Their 
L2 - curvature gradient flow method is obtained from a variational scheme 
minimizing the next energy cost functional: 


E(u)= [ £V7ul\exeay (6) 


where (2 R® and f represents an increasing function. The following fourth- 
order isotropic diffusion model is achieved after applying Euler-Lagrange 
equation and the gradient descent approach: 


Soh eee] ) 
u(O, x; y) =Uo (x, y) 
there the diffusivity function is »g(s) = A Air [18]. The You-Kaveh restoration 
1+ : 


method provides effective Gaussian noise filtering and avoids successfully the 
image staircasing but, unfortunately, it may also generate the multiplicative 
speckle noise. A despeckling algorithm is provided in [18] to deal with this issue. 
Another important nonlinear fourth-order PDE-based denoising technique is the 
LLT model of M. Lasaker, A. Lundervold and X. C. Tai [19]. It is obtained from 
the variational scheme: 


u= arg min Spr +2 (u-uy) a2 (8) 
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[He ci Jey ; . The LLT restoration 


1/2 
us) 
scheme outperforms the You-Kaveh scheme for an appropriate choice of the 1 
parameter. It produces a very good feature-preserving noise reduction and 
overcomes efficiently the staircasing effect [19]. 
We also proposed many effective variational image filtering approaches that 
remove the blurring, preserve the image features and avoid the staircasing. They 
are disseminated in some of our research papers [8, 20]. Let us present here the 
variational denoising model in [20] given by: 


u* = arg min Fw), 400 = ff olive) + o,(Vu)+F way) Ja (9) 


where 42 >O and |D7u| = ‘ ; : 
un, tun, tus, + 


where @, and @, represent 2 properly chosen regularizers. The next well-posed 
fourth-order PDE-based model is obtained from it: 


oO . 7 nO Z 
ae a divly, (Yul Vu) — BA (v: (\v |W 20) — Au _ Uy ) (10) 
u(O, x, y) =Up5 
where Wi Ws :[0, 00) —> (0,00), y,'(s) _ 1 Pi koh pig eS H. 
SOs Ss Os 


finite difference-based numerical approximation scheme is constructed for this 
anisotropic diffusion model. From the performed method comparison and found 
that our nonlinear diffusion method outperforms both the conventional and PDE- 
based denoising schemes. See some denoising method comparison results in 
Table 1 and Figure 1. 


Table 1. The average PSNR values for several variational and classic denoising techniques 


Method This Perona- Perona- TV You- Gaussian Median 
model Malik 1 Malik 2 Kaveh 
PSNR _ | 28.34(dB) | 26.23(dB) | 24.91(dB) | 21.88(dB) | 27.19(dB) | 21.37(dB) | 24.53(dB) 


3. Variational Structure-based Inpainting Approaches 


The variational structure-based inpainting techniques reconstruct the images 
affected by missing or highly deteriorated regions by solving the next generic 
energy functional minimization: 


min( (4) = R() +4 fap (4) dO] 


where the image domain Q < R’, D is the inpainting domain and the inpainting 
mask is provided by 2, =A-1q\p,4>0. 


(11) 
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g) You-Kaveh scheme 
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Figure 1. Barbara image filtered by variational and non-PDE models 


While the minimization of the regularizer R (u) containing some apriori 
information from the evolving image, is responsible for image filling task, fidelity 


term 5 J 25 (u — Up y dQ forces the minimizer to remain close enough to uy 
Q 


outside the inpainting domain [4, 21]. So, a generic variational image 
interpolation model is quite close to the generic variational denoising scheme 
described in the previous section. Any variational restoration model can be 
adapted for the reconstruction task by applying an inpainting mask. 

Various interpolation models can be obtained by considering various regularizer 


forms. The Harmonic Inpainting is based on the regularizer R(w) = J ||Vual| dxdy 
Q 


[4]. It represents a quite simple energy-based inpainting model that does not 
satisfy the connectivity principle, since it cannot interpolate succesfully along the 
image gaps, and also provides too smooth results. Other early variational image 
completion methods were using the Mumford-Shah image segmentation. Such an 
interpolation scheme uses the I - convergence approximation of the Mumford- 
Shah functional [22] and it is expressed by the next minimization: 


2 
min +f Aa,(u-uy)da+¥ | 22\vuldQ+eaf e|vz,f? + Uae dQ| (12) 
: 25 2 or Q 4¢€ 


where z_ represents the edge set signature function. This reconstruction approach 
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has a low computation complexity and a fast numerical convergence. It preserves 
the sharpness of the image edges very well. 
A very influential variational image reconstruction algorithm is Total Variation 
(TV) Inpainting of T. Chan and J. Shen [23], which is given by the minimization: 
: Ap 2 

oan, (24 Neva laarey ee) andy (13) 
where @ >O and A, =A- (i— 1 oy: TV Inpainting fills the missing zone by 
minimizing the first-order total variation while keeping close to the observed 
image in the known regions. If a low @ value is applied, then the filtering is 
directed mainly to the inpainting area. It leads to the next second-order nonlinear 
diffusion model, by using Euler-Lagrange equation and the gradient descent: 


Ou .{ Vu 
u(O, x, y) =Uy 
The model (14) inpaints successfully and achieves a good connectivity, but not for 
large image gaps. Obviously, TV Inpainting is closely related to TV Denoising, 
being obtained from it by introducing the interpolation mask. Total variation 
based inpainting models of higher orders have been also proposed in the last 
years. So, the TV Inpainting model is achieved using the second-order total 
variation as regularizer component: R(u)= f |Vruldxdy [4]. It outperforms 
Q 


TV Inpainting and produces more natural reconstructed images. TV? Inpainting 
provides also a better connectivity, being able to interpolate properly along large 
image gaps. The first and the second-order TV regularizations could be also 
combined in order to get improved interpolation results. Such a mixed TV + TV? 
Inpainting model inpaints the image using the minimization [24]: 


u. =arg min a) fu — uy) dxdy + J a(xVuldxdy + | BG@)V7uldxdy (15) 
4 2 Q\D 2 Q 


where (x) and /3(x) represent 2 spatially varying functions that control the 


completion process. This combined approach outperforms both TV and TV? 
Inpainting. It fills properly large inpainting areas, overcoming the blocky effect, 
and is solved numerically via the Split Bregman algorithm [24]. Total Generalized 
Variation (TGV), which is a generalized version of the total variation model that 
involves higher-order derivatives of u, can be also applied for inpainting [25]. 
Other improved TV-based inpainting techniques are Blind Inpainting using /o and 
TV Regularization [26] and TV Inpainting with Primal-Dual Active Set [27]. 
Another well-known higher-order variational image interpolation solution is the 
Euler’s Elastica Inpainting proposed by Chan and Shen [28]. Their reconstruction 
model uses the next regularizing function: 
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R(u)= Fein + A - ws) reese (16) 


where the coefficients @, 2 >O control the behavior of this technique and w (u) 
is a weighting function depending on the image’s histogram. Euler’s Elastica 
Inpainting executes an efficient reconstruction, being able to inpaint large image 
zones and works for noisy images too [28]. It achieves a much better connectivity 
than TV Inpainting and other TV-based methods. 

Several inpainting examples illustrating the connectivity power of various 
variational schemes are displayed in Figure 2. 


a) Harmonic Inpainting§ c) TV Inpainting (large gap) e) Euler's Elastica Inpainting 
(small gap) (large gap) 


= — = = = 
4 
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b) TV Inpainting d) TV-2 Inpainting 
(small gap) (large gap) 


— 
<<. 


Figure 2. Comparing the connectivity of several variational completion schemes 


We proposed many variational. structural inpainting techniques in our past 
research papers [8, 29]. The variational approach in [29] recoveres the affected 
image applying the minimization u,,, = arg min F(z), where 


opt 


Fw) =§{ee,(IVul)+ Fam) fo 7) 


where the inpainting domain IX — © and the regularizer function has the form 
1 


9,(S) = [re Mu) | gz, with 77(u(x, y,)) = Gmedian(|Vull)+v1 . 
0 y log ,o(s 7 7(u)) +O 


where a, BE(01), 5.7,5,v —(O,1] and S &[1,5). It leads to the 
following well-posed nonlinear anisotropic diffusion-based model: 
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O ‘ 
A = 2adivly,,(\Vul|)Vu)— BA-1,)-(u—u) 
Ot (18) 
u(0, x, y) = Uy 
u(t, x, y)=0,on OQ\T 
ES 
3 
where yw, (s) = ef (4) . Its weak solution representing the 
vlog (s+mu)y +6 


interpolation output is determined numerically using a consistent fast-converging 
finite difference method based approximation scheme [30]. This method produces 
a very good image inpainting in both normal and noisy conditions, preserving the 
boundaries and other features, and avoiding the unintended effects. The method 
comparison results are described in Table 2 and Figure 3. 


Table 2. The PSNR values obtained by some state of the art inpainting methods 


Method} Our model | Navier-Stokes Kriging TV Inpainting | TV Inpainting with 
Inpainting [31]] Interpolation [32] PDAS 


PSNR | 33.14 (dB) | 26.42 (dB) 30.38 (dB) 32.85 (dB) 33.96 (dB) 


a) Degraded image b) Our inpainting c) Navier-Stokes Inpainting 
: | 


4. Conclusions 


An overview of the energy-based image denoising and structural inpainting, 
which represent two closely related research fields, have been presented in this 
paper. The two image processing domains are based on generic energy functional 
cost minimization procedures that are closely related, since the variational 
restoration models can be adapted for interpolation by introducing an inpainting 
mask. The described variational denoising and inpainting models lead to some 
second and fourth-order nonlinear diffusion model that can be solved numerically 
by applying some iterative numerical approximation algorithms. Many restoration 


14 Tudor Barbu 


and inpainting techniques surveyed here are using total variation regularizations 
of first and higher orders. These schemes provide effective feature-preserving 
noise removal and inpainting results and overcome the undesirable image effects. 
Some of our own contributions in these areas have been also described. Our 
variational techniques achieve satisfactory denoising and reconstruction results, 
overcome properly the unintended effects and outperform many well-known PDE 
and non-PDE restoration and interpolation schemes. They could be further applied 
in other image analysis fields approached by us, such as object detection and 
recognition [33-35]. Unfortunately, our variational models perform structure- 
based interpolation only, not working properly for textures. So, the textural 
inpainting will represent the focus of our future research. 
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